feat(plugins): project LlmResponse finish_reason to BigQuery analytics#5704
Open
roanny wants to merge 2 commits into
Open
feat(plugins): project LlmResponse finish_reason to BigQuery analytics#5704roanny wants to merge 2 commits into
roanny wants to merge 2 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Projects `finish_reason` from `LlmResponse` to the `attributes` JSON column of `LLM_RESPONSE` events written to BigQuery by `BigQueryAgentAnalyticsPlugin`, and exposes it as a typed column in the generated per-event SQL view. Without this, classifying model failure modes (MAX_TOKENS, SAFETY, MALFORMED_FUNCTION_CALL, RECITATION, etc.) via SQL against the analytics table is impossible — the signal lives only in unstructured Cloud Logging output that is not joinable to the structured event stream. Changes: - `EventData` gets a `finish_reason: Optional[str]` field. - `after_model_callback` reads `llm_response.finish_reason.name` via a defensive getattr (same back-compat pattern as `cache_metadata`, covering legacy response shapes). - `_enrich_attributes` writes `attrs["finish_reason"]` only when set, keeping existing consumers and partial streaming events unaffected. - `_EVENT_VIEW_DEFS["LLM_RESPONSE"]` projects `JSON_VALUE(attributes, '$.finish_reason')` as `finish_reason`. - Schema-field description for `attributes` mentions the new key with example values. Tests: - `test_after_model_callback_projects_finish_reason` — MAX_TOKENS round-trips end-to-end into the BQ row's attributes. - `test_after_model_callback_omits_finish_reason_when_absent` — key is omitted when `LlmResponse.finish_reason` is None. - `TestEnrichAttributes::test_finish_reason_included_when_set` and `..._omitted_when_none` — direct `_enrich_attributes` coverage. - Existing 229 plugin tests still pass; 5 skipped (pre-existing). Fixes google#5644
roanny
force-pushed
the
feat/llm-response-finish-reason
branch
from
5063422 to
2198289
Compare
roanny
changed the title
Author
|
@googlebot I signed it! |
Collaborator
rohityan
added
bq
Collaborator
|
Hi @roanny , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Your PR has been received by the team and is currently under review. We will provide feedback as soon as we have an update to share. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Projects
finish_reasonfromLlmResponseto theattributesJSON column ofLLM_RESPONSEevents written to BigQuery byBigQueryAgentAnalyticsPlugin, and exposes it in the generated per-event-type SQL view.Without this projection, classifying model failure modes (MAX_TOKENS, SAFETY, MALFORMED_FUNCTION_CALL, RECITATION, etc.) via SQL against the analytics table is impossible — the signal lives only in unstructured Cloud Logging output that is not joinable to the structured event stream.
Fixes #5644.
Changes
EventDatagets afinish_reason: Optional[str]field.after_model_callbackreadsllm_response.finish_reason.nameand passes it through toEventData._enrich_attributeswritesattrs["finish_reason"]when present (omitted entirely whenNone, so the JSON shape stays backwards-compatible for existing consumers)._EVENT_VIEW_DEFS["LLM_RESPONSE"]projectsJSON_VALUE(attributes, '$.finish_reason') AS finish_reasonso it shows up as a typed column on the per-event-type view.attributescolumn mentions the new key with example values.Backwards compatibility
The new key is only added to the JSON when
LlmResponse.finish_reasonis notNone, so events emitted before this change and events for streaming partials (wherefinish_reasonis unset) remain unchanged. SQL consumers reading the existing documented fields are unaffected. New consumers can opt in viaJSON_VALUE(attributes, '$.finish_reason')or via thefinish_reasoncolumn exposed by the generated view.Test plan
New unit tests (all passing locally;
229 passed, 5 skippedfor the full plugin suite):test_after_model_callback_projects_finish_reason— assertsMAX_TOKENSround-trips end-to-end into the BQ row'sattributes.test_after_model_callback_omits_finish_reason_when_absent— asserts the key is omitted whenLlmResponse.finish_reasonisNone.TestEnrichAttributes::test_finish_reason_included_when_set— direct_enrich_attributescoverage.TestEnrichAttributes::test_finish_reason_omitted_when_none— direct_enrich_attributescoverage of the None branch.To run locally:
cc @sanketpatil06 @darshil3011